1858, from off (adv.) + color (n.); originally used of gems; figurative extension to "of questionable taste, risqué" is American English, 1867.
Somewhat salacious; risque´; blue: a couple of off-color jokes/ Some of his observations were a bit off color (1875+)